As modern applications become larger, more complex, and more dynamic, building tools to manipulate these programs becomes increasingly difficult. At the same time the need for tools to manage applications grows. Information-gathering tools are needed for program analysis, introspection, and instrumentation to aid in software development, testing, debugging, and simulation. There is also a need for tools that modify programs for optimization, translation, compatibility, sandboxing, etc.
Many modern applications are assembled and defined at runtime, making use of shared libraries, virtual functions, plug-ins, dynamically-generated code, and other dynamic mechanisms. The amount of program information available statically is shrinking. Static tools have necessarily turned to feedback from profiling runs, but these provide only an estimate of program behavior. In many cases, the complete picture of a program's runtime behavior is only available at runtime.
Consider an important modern application, the web server. Today's web servers are built for extension by third-party code, in the form of dynamically-loaded modules (e.g., Internet Server Application Programming Interface (ISAPI) components used to provide dynamic data and capabilities for web sites). Even the designers of the web server programs cannot anticipate all of the third-party code that will be executed when the web server is in actual use.
Some runtime systems that gather information about or allow for manipulation of applications make use of a code cache implemented in software. Code is placed in the code cache so that it can be used for various purposes. When executing a single application in isolation, there may be no reason to limit the size of the code cache. However, when executing many programs simultaneously, memory usage can become problematic and can be reduced by imposing a bound on the size of the code cache. However, cache bounds come with a performance cost, and the trick is to pick the bound with the best space and performance tradeoff.
Many systems with a software code cache use a hard coded size limit. When the size limit is reached, the entire cache is flushed. The limit is set generously, and it is assumed that it will rarely be reached. This may work when executing a benchmark suite, but a generous hard coded size limit is not as well suited when targeting disparate applications like desktop programs.